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Summary 


Problem 

The  ongoing  war  on  terrorism  makes  medical  surveillance  and  remote  access  to  medical 
care  for  deployed  forces  pressing  concerns.  The  potential  for  chemical  and  biological  terrorism 
on  U.S.  soil  adds  an  urgency  for  Navy  medical  technologies  to  respond  to  Homeland  Defense 
concerns,  as  well.  An  assessment  of  Navy  telemedicine  as  a  complex  healthcare  support  system 
is  needed  to  demonstrate  how  current  practices,  training,  equipment,  and  expenditures  measure 
up  to  the  emerging  needs  of  the  Fleet. 

Objective 

This  report  reviews  military  and  civilian  models  for  evaluating  telemedicine  systems  in 
order  to  determine  future  directions  for  Navy  telemedicine  research  within  the  current  funding 
environment.  How  can  we  calculate  the  level  of  technology  to  implement  for  the  most 
productive  return  on  investment  across  varying  treatment  settings? 

Approach 

A  literature  review  of  military  and  civilian  telemedicine  was  conducted  in  September 
2001  to  determine  what  types  of  evaluative  models  are  currently  being  advocated  by 
telemedicine  practitioners.  A  review  of  current  models  for  evaluating  telemedicine  yielded 
seven  categories  for  evaluation:  tools  and  equipment;  outcomes;  cost;  treatment  settings,  task 
domains;  participant  satisfaction;  and  human  factors. 

Using  this  literature  review  as  a  baseline,  the  authors  developed  a  conceptual  model  for 
assessing  the  structure  of  Navy  telemedicine.  The  model  describes  how  each  of  the  seven 
components  affect  the  implementation  of  telemedicine  in  Navy  and  civilian  settings.  Finally,  the 
authors  recommend  how  short-term,  narrowly-focused  studies  could  form  the  basis  for  a  model 
for  correlating  level  of  technology  suites  to  level  of  care  across  treatment  settings. 

Results 

An  analysis  of  models  for  evaluating  telemedicine  indicates  that  there  is  currently  no  workable 
model  for  evaluating  telemedicine  as  a  complex  and  multi-level  system,  either  in  military  or 
civilian  research.  While  military  telemedicine  circumvents  some  problems  faced  in  civilian 
systems,  issues  such  as  wait  and  queue,  connectivity,  continuum  of  care  across  variable  treatment 
sites,  and  healthcare  needs  in  deployed  platforms  make  it  difficult  accurately  to  measure  the 
effectiveness  of  telemedicine  at  different  levels  of  care. 

Conclusions 

A  model  for  evaluating  telemedicine  that  correlates  level  of  technology  to  level  of  care 
should  be  developed.  A  system  of  measurable  criteria  for  determining  the  appropriate 
telemedicine  suite  to  implement  for  a  given  population  at  risk  across  varying  treatment  facilities 
needs  to  be  designed  so  that  the  maximum  return  on  investment  can  be  achieved. 
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1.0  Introduction 


Telecommimications  technologies — long  an  integral  part  of  medical  support  for 
the  U.S.  armed  forces — have  increasingly  become  the  standard  of  support  in  serving  a 
treatment  population  at  work  in  varied  environments.  Personnel  may  be  deployed  in 
combat,  peacekeeping,  or  humanitarian  efforts  in  environments  that  are  relatively 
inaccessible  or  remote  from  conventional  military  healthcare  resources,  or  that  pose 
special  challenges  because  of  harsh  conditions  (Vidmar  1999).  The  ongoing  war  on 
terrorism  makes  medical  surveillance  and  the  ability  to  provide  immediate  medical  care 
to  these  deployed  forces  an  even  more  pressing  concern.  Given  the  enormous  investment 
the  DOD  has  made  in  implementing  telemedicine  ($327  million  between  1993-1998), 
how  can  we  determine  that  we  have  efficient  systems  in  place  to  serve  our  troops  (Reed 
2002)? 

The  Naval  Health  Research  Center  (NHRC),  San  Diego,  CA,  has  an  established 
history  of  research  evaluating  the  potential  impact  of  advanced  telecommunications  for 
shipboard  medical  departments.  Even  before  the  Navy  issued  specific  directives  to  do 
such  research  (Patel  1994;  Fisher  1994;  Tillery  1995),  Nice  (1987)  documented  Navy 
medical  communications  and  medical  evacuations  (MEDEVACs)  aboard  ships  at  sea, 
concluding  that  telecommunications  technologies  could  greatly  reduce  the  incidence  of 
medevacs  and  improve  quality  of  care  at  sea.  The  medical  personnel  interviewed  as  part 
of  that  study  believed  that  x-rays  and  image  transmission  would  have  been  the  most 
useful  forms  of  telemedicine  for  them  at  sea.  Senior  medical  personnel  estimated  that 
28%  of  medevacs  could  have  been  avoided  with  appropriate  telemedicine. 

Following  the  1994  and  1995  directives,  NHRC  researchers  continued 
investigating  and  evaluating  telemedicine  in  studies  focusing  on  equipment  and  training 
needs,  treatment  sites  (such  as  aboard  ship);  and  specialties  such  as  telemental  health  and 
Ear/Nose/Throat  (ENT).  Gauker,  Pugh,  and  Pearsall  (1995)  used  independent  duty 
corpsmen  (IDC)  rankings  of  telecommunications  technologies  for  their  utility  in  Navy 
healthcare  and  concluded  that  telemedicine  could  be  effective  if  based  in  careful 
planning,  training,  and  evaluation.  IDC  providers  surveyed  as  part  of  this  study  ranked 
basic  modes  of  communication  over  complex  ones  for  use  in  their  shipboard  medical 
problems.  In  order  of  preference,  the  technologies  were  telephone,  radio,  fax,  email, 
x-ray,  still  pictures,  and  VTC. 

Larson,  Burr,  Pearsall,  and  Silva  (1998)  analyzed  self-reports  from  Navy 
telemedicine  users  on  board  aircraft  carriers  and  found  that  shipboard  providers  perceived 
telemedicine — especially  low-end  technology — as  a  clinically  effective  tool.  This  survey 
of  medical  providers  found  that  VTC  and  Internet  based  modalities  were  frequently  used 
for  dermatology  or  orthopedic  questions.  Telemedical  consults  changed  39%  of 
treatment  plans,  a  finding  which  suggests  these  consults  have  substantial  clinical  impact. 

Lane,  Swistak,  and  Konoske  (1999)  evaluated  the  medical  workstation  (MEWS) 
during  field-testing  exercises  during  Kernel  Blitz  99  to  determine  whether  the  equipment 
increased  provider  satisfaction,  productivity,  medical  readiness,  and  clinical  capabilities 
at  far  forward  and  remote  areas  of  care.  The  MEWS  system  consists  of  small  computer 
stations  and  provides  real  time,  far  forward  medical  data  that  can  be  stored  and 
transferred.  Providers  were  generally  satisfied  with  the  system,  itself,  and  its  productivity 
for  their  care.  Power  outages  compromised  the  reliability  of  system  operation  and 
decreased  some  providers’  ratings.  Provider  comments  strongly  suggested  that  any 
system  implemented  should  support,  rather  than  change,  the  normal  workflow  of 
providers  at  far  forward  echelons  of  care  in  order  to  ensure  utilization. 
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Melcer  and  colleagues  evaluated  factors  influencing  the  development  of  a 
telemedicine  network  in  medical  treatment  facilities  (MTFs)  of  Region  9  onshore, 
focusing  on  the  clinical  impact  of  telemedicine  for  ENT  specialty  care.  In  a  retrospective 
study  (Melcer,  Crann,  Hunsaker,  Deniston,  &  Caola,  2002),  a  new  telemedicine  network 
using  VTC  for  consults  and  a  web-based  application  for  scheduling  sessions  showed 
substantial  growth  in  the  rate  of  telemedicine  use  over  a  2  year  period.  The  ENT 
specialty  showed  the  largest  increases  in  rate  of  use  relative  to  neurology  and  child 
psychiatry.  In  a  prospective  study  (Melcer,  Hunsaker,  Crann,  Caola,  &  Deniston,  2002) 
ENT  telemedicine  via  VTC  produced  substantial  clinical  impact,  with  approximately 
45%  of  consults  leading  to  a  change  in  diagnosis.  This  rate  of  diagnosis  change 
generalized  across  patient  demographic  and  type  of  ENT  condition. 

The  purpose  of  NHRC  telemedicine  studies  to  date  has  been  to  assess  the  need  for 
and  clinical  impact  of  medical  telecommunications  ashore  and  afloat,  and  to  evaluate 
whether  telemedicine  applications  utilized  in  MTFs  on  shore  might  generalize  to 
shipboard  medicine.  The  variables  examined  have  necessarily  been  limited  so  that  the 
available  data  could  provide  clear  answers  to  narrowly  defined  research  questions.  Thus, 
a  complex  system  of  healthcare  delivery  has  largely  been  studied  as  a  series  of  separate 
applications  operating  as  parallel  platforms.  A  model  has  not  yet  been  developed  to 
account  for  Navy  telemedicine  as  a  dynamic  system  that  relies  on  interoperability  among 
hardware,  software,  personnel,  and  data. 

Purpose  of  this  report 

This  literature  review  has  three  objectives: 

•  Review  models  for  evaluating  telemedicine  systems  in  the  current  literature 

•  Ascertain  key  criteria  proposed  for  assessing  telemedicine  systems 

•  Compare  military  studies  to  date  with  proposed  models  to  determine  new 
directions  for  Navy  research  in  the  current  funding  climate 

An  assessment  of  telemedicine  as  a  complex  system  is  needed  to  demonstrate  how 
current  processes,  usage,  training,  equipment,  and  expenditures  measure  up  to  the 
emergent  needs  of  the  Fleet  as  a  whole.  Assessing  telemedicine  as  an  integrated  system 
of  healthcare  delivery  would  simultaneously  account  for  organizational  and  technological 
factors  in  order  to  evaluate  how  information  gets  distributed  and  processed.  This  type  of 
analysis  would  extend  the  concept  of  technical  interoperability — the  ability  of  software 
and  hardware  to  exchange  information  and  share  tasks  and  resources — to  human  and 
organizational  structures.  The  most  important  research  focus  for  the  foreseeable  future, 
then,  would  be  to  address  how  technical  configurations  and  organizational  structures 
affect  the  flow  of  information  and  resources  in  a  given  telemedicine  system.  Evaluating 
the  degree  of  technical  interoperability  and  the  organizational  ability  to  pool  knowledge 
and  resources  in  a  system  can  help  ensure  that  that  each  system  is  deployed  to  its 
maximum  capacity. 
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Structure  of  this  report 

This  study  was  conducted  primarily  by  reviewing  telemedicine  literature  that  did 
one  of  three  things:  proposed  a  theoretical  model  for  evaluating  the  effectiveness  of 
telemedicine;  described  an  existing  telemedicine  program;  or  attempted  to  measure 
outcomes  of  key  components  of  telemedicine  systems.  Section  2.0  describes  how  the 
authors  of  this  report  selected  the  articles  reviewed  in  this  report.  Section  3.0  discusses 
the  applicability  of  civilian  models  to  military  telemedicine.  Section  4.0  discusses  seven 
key  criteria  for  assessing  telemedicine  systems  suggested  by  researchers  in  both  civilian 
and  military  telemedicine.  This  section  also  describes  programs  or  studies  that  reflect  the 
focus  and  potential  of  each  assessment  model.  Although  long-term,  rigorous  empirical 
studies  are  relatively  rare,  the  authors  have  attempted  to  provide  data-driven  studies 
wherever  available.  Section  5.0  recommends  new  directions  for  Navy  research  and 
suggests  possible  structures  for  follow-up  studies,  while  Section  6.0  sums  up  important 
shifts  in  focus  in  telemedicine  assessments. 

2.0  Method 

A  literature  review  was  conducted  in  September  2001  in  the  library  database 
PubMed  using  the  search  term  “telemedicine”  in  combination  with  “evaluation”  or 
“model.”  The  search  yielded  602  empirical  and  non-empirical  evaluations  of 
telemedicine,  69  of  which  evaluated  telemedicine  in  military  settings.  Studies  that 
assessed  specific  products  that  had  become  outdated  or  underused  were  eliminated  from 
consideration  so  that  only  models  for  evaluating  technologies  likely  to  be  in  general  use 
were  reviewed.  The  abstracts  of  the  remaining  articles  were  then  reviewed  and  sorted  by 
the  type  of  evaluation  each  performed  or  advocated  (i.e.,  cost/benefit  analysis,  provider 
satisfaction  assessment,  and  so  forth).  Technical  reports  available  through  the  NHRC 
library  and  the  Defense  Technical  Information  Center  (DTIC)  were  analyzed  using  the 
same  method. 

Since  the  articles  reviewed  in  this  search  were  largely  written  by  MDs,  a  related 
literature  search  was  conducted  via  the  PubMed  and  the  DTIC  databases  during 
September  2001  using  the  search  term  “telenursing”  to  get  a  preliminary  indication  of  the 
perceptions  of  medical  providers  such  as  nurses  and  IDCs.  Some  143  article  abstracts 
were  read  and  sorted  as  previously  described.  The  volume  of  articles  in  this  area  suggests 
the  viability  of  a  separate  study  comparing  the  experiences  of  nurses  and  IDCs.  The 
telenursing  articles  were  thus  omitted  from  this  review  in  order  to  focus  on  less 
specialized  models  for  evaluating  telemedicine.  For  the  purposes  of  this  report,  the 
telenursing  articles  would  have  formed  a  sizeable  subset  of  the  “human  factors”  category 
discussed  later  in  this  report. 

The  authors  of  this  report  designated  a  function  or  type  for  each  article  reviewed, 
as  follows. 

•  Structural  analysis  or  model  for  a  potential  program 

Articles  in  this  category  described  a  theory,  philosophy,  or  structure  that  the 

telemedicine  literature,  current  state  of  technology,  or  experience  with 

telemedicine  implemented  to  date  suggested. 
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•  Description  of  the  progress  of  an  existing  program 

Articles  in  this  category  described  the  equipment,  staffing,  logistics,  or  other  “nuts 
and  bolts”  aspects  of  a  specific  program.  Little  or  no  data  was  collected  in 
support  of  a  specific  research  question.  The  primary  focus  was  how  a  particular 
program  functioned. 

•  Empirical  analysis  of  data  from  existing  program 

Articles  in  this  category  analyze  data  from  surveys,  rate  of  equipment  usage, 
treatment  outcomes,  and  similar,  quantifiable  factors  in  an  up-and-running 
telemedicine  program.  Most  of  the  work  done  at  NHRC  falls  into  this  category. 

In  the  following  review  of  evaluative  studies  in  telemedicine,  the  function  of  each 
article  (structural,  descriptive,  or  empirical)  has  been  designated  as  well  as  the  type  or 
focus  of  the  evaluation.  Table  1  summarizes  the  types  of  models,  program  descriptions, 
and  short-term  empirical  studies  that  attempt  to  define  how  telemedicine  systems 
currently  work. 


Theoretical 

.v;l$ My:: 


Descriptions 


Limited 

Empirical 

Studies 


Outcomes  of  Literature  Review. 

The  authors  of  this  report  identified  an  extensive  number  of  evaluative  models, 
descriptions  of  the  implementation  of  specific  telemedicine  projects,  and  studies  that 
attempted  to  assess  the  efficacy  of  specific  components  in  larger  telemedicine  systems. 
Analysis  of  these  revealed  recurring  (advocated  in  more  than  five  documents)  proposals 
for  or  demonstrations  of  specific  types  of  evaluation: 

•  Assessments  of  tools  and  equipment 

•  Integration  or  outcome  studies 

•  Cost  analyses 

Although  not  as  prevalent  in  the  telemedicine  literature,  the  following  categories 
of  evaluation  were  also  advocated  as  part  of  an  integrated  evaluation  of  telemedicine 
systems: 

•  Evaluations  of  telemedicine  treatment  settings 

•  Task  domain  studies 

•  Evaluations  of  participant  satisfaction 

•  Human  factors  evaluations 


These  seven  categories  appear  to  comprise  a  consensus  in  the  literature  regarding 
which  components  should  be  assessed  in  evaluating  the  structure  of  telemedicine  as  a 
system.  The  authors  of  this  report  read  model  proposals,  program  descriptions,  and  short¬ 
term  empirical  studies  (when  available)  from  each  of  these  seven  categories  and 
summarized  examples  representing  each  type  of  evaluation  for  inclusion  in  this  report.  In 
addition  to  these  studies  of  the  individual  components  of  telemedicine  systems,  the 
authors  reviewed  articles  that  attempted  to  construct  broad-based  models  for  evaluating 
how  these  components  work  together  as  a  complex  system.  A  working  list  of  criteria  for 
evaluating  telemedicine  as  a  complex  system  was  constructed  for  the  purposes  of  this 
report  by  combining  the  comprehensive  model  suggested  by  Yawn  (2000)  with  human 
factors  criteria  advocated  in  Yellowlees  (1997, 1998). 

Table  2  lists  the  criteria  for  evaluating  telemedicine  systems  that  have  been  repeatedly 
proposed  in  the  literature  and  summarizes  representative  studies  in  each  of  the  seven 
categories  in  order  to  suggest  their  applicability  to  military  telemedicine. 
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Task  Domain 

Evaluative  Model 

Grigsby  et  al.  (1995) 

List  of  healthcare  processes  for  evaluation 
(e.g.,  patient  education,  urgent  evaluation) 

Military  Utility 

Limited. 

May  be  useful  when 
developing  a  technology. 

Empirical  Study 

Potter  et  al.  (1997) 

Derm  specialty  most  used. 

Evacs  decreased  as  telemedicine  increased 
in  data  based  projections. 

Program 

Description 

No  relevant  references 

N/A 

Tools  and  Equipment 

Evaluative  Model 

Ganguly  and  Ray  (2000) 

Interoperability  of  software  systems  at 
Various  levels  is  key  to  optimizing 
telemedical  potential 

Military  Utility 

Extensive. 

Both  human/machine 
interface  and  component 
“interoperability”  (e.g., 
linkage  of  disparate 
databases  across  MTFs, 
specialties  and  remote 
treatment  sites)  are  critical 
for  effective  operation. 

Empirical  Study 

Mun,  Levine,  Cleary  & 

Dai  (1998) 

Deployable  teleradiology  supported  full 
MASH  workload  in  Bosnian  theatre  and 
appears  effective  in  reducing  evacuations. 

Program 

Description 

!  Birkraire-Peters,  Peters,  & 
Whitaker  (1999). 

Procedures  for  evaluating  human  factors 
::  and  usability  proposed. 

Telemedicine  Settings 

Evaluative  Model 

•'  1)  Beach,  Miller  &  .V;>- 
Goodall  (2001) 

The  cost-effectiveness  of  telemedicine  may 
be  specific  to  the  treatment  setting. 

2)  Filiberli,  Wallace, 
Kateeswaran  &  Neft 
(1995) 

;  A  telemedicine  transaction  model  (TIM) 
can  track  the  path  of  medical  interventions 
from  presentation  to  solution. 

Military  Utility 

Extensive. 

Evaluation  of  deployed 
treatment  sites  such  as 
different  ship  types  would 
permit  effective  adaptation 
of  telemedicine  capabilities. 

Empirical  Study 

Whitlock  et  al.,  (2000) 

Remote  home  monitoring  of  diabetics 
reduced  physiological  risk  measures. 

Best  Available  Copy 


Sisfegration/Ositcome 


EvaluativeModel  Grigsgby  et  al.,  1995 


Military  Utility 


Empirical  ^  Roine  et  al.  (2Q01) 
Study 


A  model  predicting  the 
conditions  where 
telemedicine  would  be  most 
useful  could  measure 
potential  impact  in  a 
deployed  setting. 


Program 

Description 


2)  Walters  et  al.  (1996) 


Burgess  et  al.  (1999) 


Cost  Analysis 


Military  Utility 
Moderate. 

The  military  mission  to 
provide  care  to  troops 
anywhere  supercedes  cost 
factors.  However,  cost 
analysis  is  important  in 
evaluating  telemedicine  as 
an  alternative  to 
MEDEVACS. 


Evaluative  Model  Cameron  et  al.  (1998) 


Empirical  Study  Stoloffet  al.  (1998) 


Program 

Description 


None 


Telemedicine  impact  determined  by 
incidence  of  medical  problem,  diagnostic 
accuracy,  potential  for  improving  patient 
condition. 


1)  Metaanalysis  finds  few  strong 
evaluation  studies.  Existing  data  supports 
effectiveness  of  teleradiology. 


2)  70%  of  telemedicine  consults  impacted 
patient  status. 


Development  of  telemedicine  programs 
should  follow  needs  assessment  from 
patient  population 


Simulations  used  to  project  variables  that 
lead  to  cost  saving.  Frequency  of  use  was 
most  important. 


Store  and  Forward  telemedicine  projected 
to  be  ciost  effective  for  all  Navy  ship  types. 
Teleradiology  and  VTC  for  large  ships 
Only,  K  y;.,:.', 


Best  Available  Copy 


.■ifv-.-'-  gy-.;*fcv 

flliliil!®! 

Participant  SatisfaetloM 

Evaluative  Model 

Review  of  32  empirical  patient  satisfaction 
studies  found  current  research  models 
lacking  in  rigor. 

Military  Utility 

Limited. 

Feedback  on  technology 
usability  and 

provider/patient  preferences 
may  help  predict 
telemedicine  use. 

Empirical  Study 

Sixsmith  (2000) 

Home  monitoring  encouraged 
help  seeking  by  patients  despite 
technical  pitfalls. 

Program 

Description 

May  et  al  (2001) 

Providers  who  attempted  to  make 
telemedicine  technologies  fit  their  own 
standard  practices  failed  to  implement 
telemedicine  successfully. 

Human  Factors 

Evaluation 

Evaluative  Model 

Yellowless  (1997,  1998) 

Provider  investment  in  telemedicine 
technologies  is  critical  for  program 
success. 

Military  Utility 

Empirical  Study 

N/A 

Extensive. 

Important  for  understanding 
how  new  telemedicine 
initiatives  can  be  successful 
and  for  ensuring  effective 
use  of  technologies  in 
various  settings. 

Human  Factors  V 

assessments  are  often 
combined  with  other 
types  of  evaluations. 
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Program 

Description 

;n/a:;  V  V 

3.0  Applicability  of  Civilian  Studies 


Telemedicine  is  the  use  of  telecommunications  technologies  such  as 
videoteleconferencing  (VTC),  the  Internet,  email,  and  telephone  to  assist  in  delivering 
healthcare  to  remote  or  distant  treatment  sites  (Bashshur  1995;  Grigsby,  Schlenker, 
Kaehny,  Shaughnessy,  &  Sandberg  1995;  Jerome  et  al.  2000).  Reviews  of  the  field’s 
brief  (30  year)  history  note  that  telemedicine  systems  evolve  over  time,  vary  across 
specialties,  and  require  different  evaluative  foci  at  different  stages  of  their  development 
(Bashshur,  Reardon  &  Shannon  2000). 

As  innovations  in  telecommunications  make  remote  care  more  feasible  and  as 
systems  mature,  both  military  and  civilian  studies  have  begun  advocating  the 
development  of  a  flexible  model  for  evaluating  more  complex  systems  of  remote 
healthcare  delivery.  Attempts  to  produce  a  model  for  systemic  evaluation  have  called  for 
standardization  of  evaluative  criteria  in  the  telemedicine  field,  but  no  model  to  date  has 
been  flexible  enough  to  account  for  different  practices  across  factors  such  as  location, 
duration  of  system  operation,  different  patient  populations  or  medical  specialties,  and 
human-machine  interface.  Early  (pre-1995)  studies  isolated  specific  factors  to  evaluate, 
either  in  anticipation  of  implementing  a  telemedicine  system,  or  in  assessing  evolving 
systems  in  die  early  stages  of  their  implementation.  Most  studies  evaluated  factors  such 
as  cost,  usability,  patient  satisfaction,  access,  usage,  or  the  selection,  function,  and 
performance  of  equipment  in  an  effort  to  assess  whether  telemedicine  would  prove  to  be 
a  viable  healthcare  option  in  a  given  scenario.  However,  a  workable  global  or  systemic 
approach  has  been  elusive  for  both  military  and  civilian  researchers. 

Evaluations  of  civilian  telemedicine  have  often  focused  on  telemedicine 
applications  to  rural  treatment  populations  or  prison  populations,  in  part  because  early 
funding  for  telemedicine  targeted  those  populations.  Some  recommendations  from  these 
studies  may  generalize  to  Navy  telemedicine,  since  factors  such  as  isolation  (rural)  or 
restricted  movement  or  access  to  care  (prison)  can  be  characteristic  of  deployed  military 
personnel,  as  well.  As  previously  mentioned,  another  large  body  of  literature  describes 
applications  of  telemedicine  to  nursing,  focusing  on  human  factors  that  may  parallel  the 
experiences  of  Navy  personnel  such  as  IDCs,  who  provide  healthcare  on  deployment,  but 
don’t  possess  MD  degrees. 

Bashshur  (1995, 1998)  raised  several  challenges  for  evaluating  practices  based  on 
rapidly  changing  technologies: 

•  Variable  costs  and  capabilities  of  telemedicine  systems 

•  Limitations  in  current  technologies  (rather  than  in  the  practice  of 
telemedicine  itself) 

•  The  tendency  of  studies  to  overgeneralize  results  from  specific  technologies  to 
other,  unrelated  applications 

•  Midcourse  changes  made  in  operating  parameters  during  the  period  of  a  study. 

Bashshur  concluded  that  a  financially  stable  environment,  regional  networks,  and 
the  ability  to  redistribute  resources  in  response  to  change  should  all  be  present  if 
evaluation  of  a  system  is  to  be  useful.  Clearly,  military  healthcare  is  well  situation  to 
fulfill  all  of  these  capabilities. 
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Much  like  military  telemedicine  studies,  the  civilian  literature  assesses  local 
implementations  or  specialty  applications  of  telemedicine  (Brecht,  Gray,  Peterson,  & 
Youngblood  1996;  Mekhjian,  Warisse,  Gailium,  &  McCain  1996;  Lambrecht  1997; 
McCue  et  al.  1997;  Houston  and  Rupp  2000;  Zaylor,  Whitten  and  Kingsley  2000). 
However,  an  important  recurring  theme  in  civilian  telemedicine  literature  is  the  call  for 
the  development  of  specific  categories  for  evaluating  telemedicine  as  a  comprehensive 
information  system. 

Military  telemedicine  sidesteps  some  of  the  main  concerns  of  civilian 
applications,  such  as  medical  licensing  or  legal  issues  (Grigsby  et  al.  1995;  Stanberry 
2001));  state  regulatory  policies  or  political  support  (Weissert  and  Silberman  1996; 
Lipson  and  Henderson  1996);  untrustworthy  sources  of  information  (Eysenbach  & 
Diepgen  1998);  the  corporate  or  commercial  structure  of  health  care  (Kleinke  2000; 
Shortliffe  2000;  Sinha  2000;  Jennett  and  Andruchuk  2001);  community  funding 
(Robinson  2000);  unequal  access  to  care  for  individual  patients  (Brodie,  et  al.  2000);  and 
the  effect  of  telemedicine  on  general  practice  (Thomett  2001). 

Most  military  evaluations  of  telemedicine  have  focused  on  cost/benefit 
projections  for  developing  a  system  of  telemedicine,  or  on  the  implementation  of 
telemedicine  in  a  particular  specialty  or  treatment  setting.  Tripler  Army  Medical  Center 
in  Hawaii  has  been  especially  prolific  in  producing  telemedicine  studies  that  deal  with 
different  treatment  populations  (Delaplain,  Lindborg,  Norton,  &  Hastings  1993;  Cook, 
Hansen,  Leckie,  &  Francoise  1995;  Calcagni  et  al.  1996;  Norton  et  al.  1996,  1997;  Peters 
and  Peters  1998;  Birkmire-Peters,  Peters,  &  Whitaker  1999;  Burgess  et  al.  1999; 
Garshnek  and  Hassell,  1999;  and  Hill,  Allman,  and  Ditzler  2001.)  The  need  for  a 
framework  within  which  to  interpret  and  utilize  the  Navy’s  own  ongoing  assessment  is 
readily  apparent  (Garshnek  and  Hassell  1999). 

A  composite  model  that  combines  several  criteria  suggested  in  the  civilian 
literature  may  be  malleable  enough  to  generalize  across  applications  of  telemedicine  and 
may  be  further  adapted  to  specific  Navy  applications.  Further  study  is  needed  to 
determine  whether  each  of  these  categories,  or  a  combination  of  several  of  them,  is 
relevant  to  military  telemedicine. 


4.0  Criteria  for  Assessing  Telemedicine  Systems 

This  section  reviews  representative  models,  program  descriptions,  and  empirical 
studies  (if  available)  performed  or  proposed  within  modified  versions  of  each  of  Yawn’s 
categories,  adding  a  human  factors  component  that  many  telenursing  and  telemental 
health  studies  propose. 

4.1  Task  Domain  Studies 

Task  domain  studies  focus  on  the  sensory  perceptions  of  the  user  or  the 
discrete  tasks  that  make  up  complex  processes.  Visual  and  auditory  components 
of  medical  tasks  may  be  assessed  in  both  conventional,  face-to-face  interactions 
and  in  interactions  augmented  with  telemedicine  equipment. 

One  of  two  structural  models  proposed  by  Grigsby  et  al.  (1995)  is  an 
example  of  a  task  domain  study.  Grigsby  et  al.  evaluated  the  processes  by  which 
a  telemedicine  system  is  integrated  into  traditional  healthcare.  This  study 
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evaluated  the  costs,  access,  provider  acceptance  and  provider  use  of  telemedicine 
applications  by  types  of  process: 

•  Urgent  evaluation 

•  Surgical  follow-up 

•  Primary  care  consults 

•  Second  opinions 

•  Transmission  of  diagnostic  images 

•  Workups  or  medical  data 

•  Management  of  chronic  diseases 

•  Patient  education 

Grigsby  concluded  that  classifying  tasks  allowed  a  more  targeted 
evaluation  of  specific  processes  so  that  the  data  generated  are  more  likely  to 
generalize  to  other,  similar  tasks  across  specialties. 

Military  Studies  of  Task  Domains 

Potter,  Zdyb,  Smith,  &  Phillips  (1997)  evaluated  data  emerging  from  the  U.S. 
Army’s  use  of  telemedicine  for  the  civilian  treatment  population  in  Bosnia  in 
1995  as  part  of  a  peacekeeping  effort  connecting  MASH  units  to  Walter  Reed 
AMC  and  NMCSD.  The  authors  stressed  the  need  for  training  and  technical  field 
support  and  proposed  a  task-based  model  for  assessing  the  effectiveness  of 
deployed  telemedicine  in  peacetime.  The  Health  Care  Complex  Model  analyzes 
healthcare  “episodes”  using  five  interactive  modules: 

•  Demand  Module  -  predicts  requests  for  services 

•  Clinical  Decision  Module  -  applies  clinical  protocols  to  direct  the  flow  of 
patients  to  service  units 

•  Service  Unit  Capability  Module  -  determines  the  capabilities  of  the 
combination  of  healthcare  workers,  technologies,  and  physical 
environments 

•  Service  Unit  Operations  Module  -  assess  the  actual  tasks  providers 
perform 

•  Output  Module  -  analyzes  the  effectiveness  of  the  overall  system  in  terms 
of  cost,  patient  outcomes,  and  access  to  care. 

Each  module  is  further  divided  into  several  discrete  tasks  or  areas  of 
analysis,  so  that  key  components  in  a  complex  system  of  healthcare  can  be 
assessed  according  to  the  tasks  each  performs  in  support  of  the  whole.  VTC, 
teleradiology,  email  and  ultrasound  were  all  used.  Eighty-one  (81)  telemedicine 
consults  were  conducted,  the  majority  in  dermatology  (24.7%)  and  physical 
therapy  (19.6%).  Assessing  data  via  this  predictive  model  indicated  that  as 
telemedicine  use  increased,  evacuations  should  decrease.  The  authors  conclude 
that  standard  operating  procedures  should  include  careful  organization  of 
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technical  support  and  telemedicine  training  to  fit  field  medical  business  practices. 
This  model  may  have  application  for  telenursing  possibilities. 

Summary 

Task  domain  studies  may  not  be  a  helpful  category  for  assessment  in 
isolation  from  other  components  such  as  equipment  utilization  or  human 
factors/training.  However,  task  domain  studies  that  focus  on  sensory  perceptions 
can  help  identify  training  needs  and  describe  the  operation  of  telemedicine 
equipment  to  maximize  return  on  investment.  In  addition,  task  domain  studies 
that  examine  complex  processes,  rather  than  separate  tasks,  allow  the  separate 
components  of  a  complex  system  to  be  assessed  and  may  be  useful  in  assessing 
technologies  under  development. 


4.2  Assessments  of  Tools  and  Equipment 

Equipment  assessments  analyze  technical  requirements,  technical  support, 
patterns  of  use,  and  performance  of  telemedicine  equipment.  Many  studies 
combine  equipment  assessments  with  other  types  of  studies,  such  as  cost  analyses 
or  patient  satisfaction  surveys. 

Ganguly  and  Ray  (2000)  evaluated  models  for  interoperability--  the  ability 
of  software  and  hardware  to  exchange  information  and  share  tasks  and  resources— 
in  telemedicine  software  across  treatment  sites.  Different  software  applications  or 
hardware  systems  may  have  been  purchased  to  deal  with  specific  needs — tracking 
pharmacy  inventory,  doing  electrocardiograms  in  a  moving  ambulance,  patient 
billing,  or  archiving  provider  notes  on  patients.  The  study  discusses  increasingly 
sophisticated  levels  of  interoperability,  from  basic  physical  interoperability,  in 
which  data  can  be  manually  transferred  from  one  application  to  another,  to 
semantic  interoperability,  in  which  a  Knowledge  Interchange  Format  (KIF)  is 
designed  to  interchange  knowledge  among  disparate  software  programs.  Agent 
interoperability  (an  emerging  capability)  would  allow  mobile  software  agents  to 
move  from  host  to  host  and  to  interact  with  other  software  agents. 

Anogianakis  and  Maglavera  (1998)  and  Anogianakis  et  al.  (1998) 
provided  a  detailed  description  of  the  equipment  configuration  for  MERMAID,  a 
EU  financed,  civilian  project  that  provides  telemedicine  services  to  ships  at  sea  in 
medical  emergencies.  Paramedics  aboard  merchant  marine  vessels  follow 
medical  guidelines  based  on  the  World  Health  Organization’s  International 
Medical  Guide  for  Ships  and  utilize  INMARSAT  land-earth  station  links  to 
medical  teleconsultants.  MERMAID  combines  mobile  satellite  and  VS  AT 
technologies  and  Integrated  Services  Digital  Network  (ISDN)  protocols  to 
provide  global  reach  for  marine  telemedicine  consults.  The  authors  conclude  that 
this  combination  of  technologies  comprises  a  global  state-of-the-art  system  for 
providing  quality  care  at  sea. 
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Equipment  Assessments  in  the  Military 


At  Tripler  Army  Medical  Center,  Hawaii,  commercially  available 
telemedicine  technology  was  assessed  utilizing  human  factors  usability  criteria, 
such  as  technical  acceptability,  operational  effectiveness,  and  clinical 
appropriateness  in  performing  otological  follow-up  (Birkmire-Peters,  Peters,  & 
Whitaker  1999).  User  requirements  in  ENT  and  audiology  exams  were  measured 
against  the  technical  specifications  of  each  product  using  direct  observation  of 
provider  behavior  and  interviews  with  medical  users.  Training  needs  were 
assessed  by  videotaping  clinicians  performing  required  tasks,  and  analyzing  task 
time,  error  rate,  and  user  preferences.  Interpretations  of  digital  images  captured 
with  a  video-otoscope  were  compared  to  examinations  with  a  hand-held  otoscope 
to  assess  the  reliability,  validity,  and  appropriateness  of  telemedicine  equipment. 
Preliminary  data  indicated  that  procedures  are  sensitive  to  differences  between 
testing  instruments  regarding  the  types  of  errors  generated  and  the  time  taken  to 
complete  the  exam. 

Carlos  and  Pangelinan  (1999)  described  assessments  of  telemedicine 
technologies  performed  by  the  Telemedicine  Working  Group  in  Tricare  Region 
10.  The  group  focused  initial  test  and  evaluation  efforts  on  telepathology  and 
teledermatology,  but  found  little  user  buy-in  for  telepathology.  The  current  report 
is  a  descriptive  summary  of  tests  on  specific  products  used  in  teledermatology, 
and  the  establishment  of  working  relationships  among  users  and  technicians. 
Based  on  their  experience  with  testing  and  evaluating  teledermatology 
technologies,  the  authors  of  this  study  strongly  advocated  getting  physicians 
directly  involved  in  planning  for  and  implementing  telemedicine  in  their  facilities. 

An  empirical  study  by  Mun,  Levine,  Cleary,  &  Dai  (1998)  assessed  data 
from  the  deployable  teleradiology  (DEPRAD)  system  installed  in  support  of 
Primetime  III  to  link  MTFs  in  Bosnia  and  Hungary  during  1995.  The  article 
examined  the  system  configuration,  integration  and  support,  and 
telecommunications  network  that  enabled  the  use  of  filmless  teleradiology  in  over 
10,000  radiological  examinations  during  a  15-month  period.  This  preliminary 
assessment  outlined  the  success  of  implementing  DEPRAD  and  indicated  the 
feasibility  of  using  teleradiology  to  support  a  full  MASH  workload  in  a  deployed 
setting  for  the  first  time.  Other  advantages  included  avoiding  land-based 
evacuations  over  hostile  terrain.  The  disadvantages  included  considerable 
technical  support  early  in  the  deployment  and  substantial  transmission  time  (at 
least  a  few  minutes),  but  this  problem  was  expected  to  be  worked  out. 

A  projected  structural  model  by  W.J.  Chimiak,  Rainer,  J.M.  Chimiak,  and 
Martinez  (1997)  examined  the  concepts  and  technologies  needed  to  establish  a 
telemedicine  system  for  the  entire  Navy  Fleet.  The  article  detailed  the  structure 
and  interrelationships  among  components  of  Navy  telemedicine  architecture  and 
discussed  a  possible  implementation  scenario  to  promote  remote  medical  care  in 
the  Fleet.  Gomez,  Karinch,  and  Zajtchuk  (1996)  described  telemedicine  support 
for  humanitarian  missions  in  Somalia,  Croatia,  Macedonia,  Germany,  Italy, 
Kuwait,  Ivory  Coast,  Egypt,  Panama,  Virgin  Islands,  Kenya,  and  Haiti  from 
Walter  Reed  Army  Medical  Center  (WRAMC)  in  Washington,  DC.  The  study 
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also  analyzed  data  from  clinical  consults  from  February  1993  through  February 
1996  at  WRAMC  to  assess  the  responsiveness  of  the  telemedicine  service.  The 
study  found  that  off-the-shelf  equipment  and  low-bandwidth  transmission  could 
support  multiple  remote  treatment  sites  simultaneously.  The  utility  of  digital  still 
images  with  electronic  medical  records  was  also  demonstrated. 

Calcagni  et  al.  (1996)  examined  the  structural  model  of  Phase  I  of  the 
Primetime  III  system,  which  provided  telemedicine  support  to  the  U.S.  Army 
during  Operation  Joint  Endeavor  in  Bosnia.  The  article  used  case  reports  to  show 
the  feasibility  and  effectiveness  of  telemedicine  with  shipboard  and  US-based 
medical  departments.  The  majority  of  the  consults  assessed  were  in  radiology; 
however,  VTCs  for  ENT  and  dermatological  problems  were  also  examined. 
Several  case  reports  show  the  clinical  effectiveness  of  the  consults  and  the 
importance  of  telemedicine  training  prior  to  deployment  in  the  field.  While  this 
phase  focused  on  pinpointing  and  solving  technical  glitches  in  the  telemedicine 
system,  the  authors  emphasized  that  human  factors  are  a  strong  component  of  the 
overall  success  or  failure  of  any  implementation  of  technology.  They 
recommended  ongoing,  standardized  training,  logistical  support,  and  hands-on 
management  of  the  organizational  aspects  of  telemedicine  to  promote  a 
sustainable  telemedicine  effort. 

Summary  x 

One  of  the  most  important  research  foci  for  the  foreseeable  future  will  be 
to  address  how  technical  configurations  and  organizational  structures  affect  the 
flow  of  information  and  resources  in  a  given  telemedicine  system.  Both 
human/machine  interface  and  component  “interoperability”  (e.g.,  linkage  of 
disparate  databases  across  MTFs,  specialties  and  remote  treatment  sites)  are 
critical  for  effective  operation.  Interoperability  of  software  systems  at  various 
levels  is  key  to  optimizing  telemedical  potential.  Evaluating  the  degree  of 
technical  interoperability  and  the  organization’s  ability  to  pool  knowledge  and 
resources  can  help  ensure  that  each  system  is  deployed  to  its  maximum  capacity. 


4.3  Evaluations  of  Telemedicine  Settings 

Although  Yawn  defined  setting  as  primarily  physical,  evaluations  of 
telemedicine  settings  can  also  focus  on  the  social  environment  in  which 
telemedicine  is  implemented.  Physical  assessments  focus  on  material  setup, 
technical  skills,  and  user  training.  These  assessments  are  often  combined  with 
cost  analyses.  Social  models  evaluate  the  human  transactions  or  the  social 
context  in  which  medical  care  takes  place. 

Beach,  Miller,  and  Goodall  (2001)  proposed  an  evaluation  model  that 
assesses  costs,  patient  satisfaction,  and  equipment  (VTC  and  store-and-forward) 
in  an  accident  and  emergency  setting  in  the  United  Kingdom.  This  research 
combined  a  prospective  case-control  study  with  surveys  of  clinician,  clinical  staff, 
and  patients  to  document  treatment  outcomes,  resource  allocations,  and  patient 
and  staff  willingness  to  use  teleconsultations.  Preliminary  results  indicated  that 
the  cost-effectiveness  of  a  telemedicine  system  may  be  “situation-specific”;  that 
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is,  dependent  on  the  social  and  clinical  setting  in  which  care  takes  place.  Costs 
may  change  across  specialties,  treatment  sites,  patient  population,  and  clinical 
staff.  The  success  of  a  telemedicine  system  requires  collaboration  among  key 
users  of  the  system:  technical  staff,  administrators,  and  users. 

In  the  evaluation  model  proposed  by  Filiberti,  Wallace,  Kateeswaran,  and 
Neft  (1995),  a  Telemedicine  Transaction  Model  (TTM)  is  described  in  terms  of 
sites,  personnel,  and  events.  The  model  defines  the  elements  of  a  medical 
intervention  through  the  telemedicine  system  from  patient  presentation-^  data-> 
consultant-^  data->  disposition  of  the  problem.  For  example,  a  patient  might 
present  at  the  primary  exam  with  a  particular  set  of  symptoms,  undergo  labwork 
immediately  relayed  to  a  consultant  via  VTC,  and  receive  a  consensus  diagnosis 
and  treatment  plan  worked  out  between  the  primary  physician  and  the 
specialist/consultant  via  VTC  over  a  short  period  of  time.  Since  the  medical 
transaction  is  inherent  in  all  applications  of  both  conventional  medicine  and 
telemedicine,  the  model  can  be  used  at  different  levels  of  specificity — patient, 
program,  site,  condition,  procedure,  or  outcome.  Findings  are  easily  understood 
across  personnel  functioning  within  the  telemedicine  system  (administrators, 
providers,  technicians,  and  other  users). 


Military  Studies  of  Treatment  Settings 

In  the  Inpatient  Psychiatry  Unit  at  Tripler  Army  Medical  Center,  Hawaii, 
the  utility  of  using  VTC  to  bring  geographically-remote  family  members  into 
dialogue  with  patients  was  examined  (Hill,  Allman,  and  Ditzler  2001).  Family 
members  were  linked  with  patients  and  their  therapists  via  the  VTC  system 
sponsored  by  the  U.S.  Army  Medical  Information  System  and  Services  Agency. 
Social  transactions  among  family  members  during  VTC  therapy  sessions  were 
described  for  two  illustrative  cases.  Positive  therapeutic  outcomes  were  reported 
after  VTC  enabled  patients  to  see  and  receive  support  from  geographically  remote 
family  members.  These  case  studies  showed  the  feasibility  of  VTC  consults  and 
indicate  that  these  sessions  might  have  positive  effects  on  patient  disposition. 

Clement,  Brooks,  Dean,  and  Galaz  (2001)  assessed  outcomes  at  a 
telemedicine  neuropsychology  clinic  that  linked  Brooke  Army  Medical  Center, 
Fort  Sam  Houston,  TX,  with  Army  community  hospitals.  Initial  assessments  of 
patients  with  neurological  disorders  or  brain  injuries  were  completed  at  the 
medical  center,  while  follow-up  visits  occurred  at  local  Army  hospitals  via 
teleconferencing.  Between  mid-September  1998  and  mid-May  2000,  32  patients 
were  seen  in  87  videoconferencing  sessions.  Several  cases  were  discussed  in 
which  telemedicine  consultation  affected  patient  outcome.  Telemedicine  made  it 
possible  to  determine  whether  soldiers  with  recent  brain  injury  were  fit  for  return 
to  duty  or  should  be  separated  from  deployment.  The  authors  noted  that  military 
telemedicine  may  serve  as  a  model  for  the  civilian  community  as  well,  having 
already  dealt  with  issues  such  as  informed  consent,  confidentiality,  and  medical 
practice  across  state  lines. 

Whitlock  et  al.  (2000)  performed  a  controlled  study  of  home  telemedicine 
consultation  usipg  nurse  case  managers  in  the  treatment  of  diabetes  patients  in 
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clinics  at  the  Eisenhower  Army  Medical  Center.  Patients  in  the  telemedicine 
group  attended  diabetic  education  classes  at  the  medical  center,  and  experienced 
weekly  telemonitoring  visits  by  a  case  manager,  during  which  blood  glucose 
levels,  weight,  blood  pressure,  hypoglycemic  episodes,  exercise  and  nutrition 
goals,  and  well-being  were  reviewed.  Physicians  reviewed  the  patients  once  a 
month  via  teleconferencing,  and  the  case  manager,  family  practitioner,  and 
internist  communicated  via  email  throughout  the  study.  Patients  in  the  control 
group  were  encouraged  to  attend  the  training  sessions  but  given  routine  care  for 
their  condition.  The  study  found  that  patients  using  telemedicine  showed 
significant  improvement  in  three  key  diabetic  indicators:  ADA,  HbAic,  and  total 
body  weight. 

Hunter  et  al.  (1999)  performed  a  descriptive  comparison  of  two 
teleoncology  systems  to  outline  the  advantages  and  disadvantages  of  two 
technical  infrastructures.  The  Pacific  Oncology  Outreach  Project,  an  Internet- 
based  system  at  Tripler  for  the  most  part  used  still  pictures  and  live  audio.  Its 
goal  was  to  avoid  evacuations  from  outlying  islands  for  healthcare.  The  Region 
10  Integrated  Cancer  Network  primarily  utilized  VTC,  with  the  goal  of  promoting 
distance  learning  and  collaboration  among  medical  personnel.  Technical  aspects, 
participant  satisfaction,  and  conference  format  were  analyzed  in  both  systems. 

The  study  found  that,  while  both  systems  were  successful  and  well  accepted  by 
providers,  each  system  had  distinct  advantages.  The  Internet-based  system  at 
Tripler  allowed  users  greater  access,  and  the  ISDN-based  system  minimized 
administrative  tasks. 

Cubano  et  al.  (1999)  examined  outcomes  of  laparoscopic  procedures 
telementored  aboard  the  USS  Abraham  Lincoln.  This  was  the  first  demonstration 
of  this  kind  aboard  a  combat  ship.  The  Battlegroup  Telemedicine  system  is 
configured  to  link  the  USS  Abraham  Lincoln,  the  Johns  Hopkins  Applied  Physics 
Lab,  and  Naval  Medical  Center  at  San  Diego  (NMCSD)  via  intraship,  ship-to- 
ship,  and  ship-to-shore  modalities.  The  authors  discussed  5  cases  in  which  ship- 
to-shore  telementoring  was  used  successfully  in  performing  laparoscopic 
hemiorraphies.  This  paper  shows  the  clinical  feasibility  of  using  telemedicine  for 
mentoring  surgery  by  inexperienced  providers  in  remote  settings  such  as 
shipboard  medical  departments.  Video  links  between  the  USS  Abraham  Lincoln 
and  US-based  naval  hospitals  such  as  NNMC  and  NMCSD  allowed  5 
laparoscopic  surgeries  to  be  successfully  conducted  with  remote  guidance  by 
specialists.  The  study  results  suggested  that  successful  telementoring  depends  on 
the  working  relationship  between  operating  surgeons  even  more  than  on  the 
hardware-software  configuration. 

A  descriptive  summary  by  Vidmar  (1999)  discussed  the  principles  and 
history  of  teledermatology  in  the  DoD.  The  article  discussed  evolving  uses  of 
telecommunications  technologies  in  healthcare  and  briefly  summarized  the 
telemedicine  experiences  at  several  DoD  sites,  noting  provider  attitudes,  problems 
with  equipment,  and  staffing/training  issues.  The  article  concluded  with  a 
discussion  of  DoD  sites  that  are  evaluating  the  use  of  the  World  Wide  Web  for 
teledermatology  and  predicted  increased  use  of  teledermatology.  This  paper  also 
reviewed  evidence  that  store  and  forward  teledermatology  with  high  quality 
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digital  pictures  is  as  good  as  live  VTC  for  clinical  impact.  The  store  and  forward 
modality  also  has  many  advantages  for  scheduling  and  cost.  The  author  stressed 
the  need  for  systematic  study  of  clinical  impact  rather  than  supporting  anecdotal 
reports  seen  in  many  other  studies. 

Program  descriptions  by  Bailey  (1998)  and  a  Website  sponsored  by 
Bakalar  (1998)  outlined  the  Multimedia  Integrated  Distributed  Network  (MIDN), 
a  telemedicine  initiative  that  linked  two  carrier  battle  groups,  MTFs,  and  regional 
and  remote  clinics.  Both  sources  reported  the  configuration  of  equipment  and 
cited  successful  results  of  cases  using  telemedicine  during  the  pilot  phase. 

Norton  et  al.  (1996)  described  the  early  experiences  of  the  telemedicine 
network  established  among  the  Tripler  Army  Medical  Center,  non-government 
agencies  such  as  the  University  of  Hawaii,  and  remote  civilian  treatment 
populations  in  Micronesia.  This  description  encompasses  applications  reported  in 
earlier  articles  and  adds  a  description  of  additional  treatment  sites.  The  authors 
reported  the  establishment  of  a  successful  clinical  consultation  and  health 
education  network  using  low-bandwidth  equipment  and  primarily  preexisting 
communications  systems.  The  consults  may  be  useful  for  educational  and 
professional  interactions  as  well  as  clinical  care. 

A  program  description  by  Cook  et  al.  (1995)  briefly  outlined  the  Tripler 
Army  Medical  Center’s  implementation  of  digital  communications  in 
teleradiology  in  the  Pacific  region  using  a  “hub-and-spokes”  model.  Operation 
“Shooting  Star”  airlifts  a  Deployable  Telepresence  Unit  (DTU)  to  remote 
locations  and  connects  primary  care  providers  and  patients  with  Tripler’ s  medical 
specialists  via  satellite  or  ground  communications.  Filmless  radiology  technology 
simplifies  logistics  and  allows  remote  care  in  even  harsh  environments.  Delaplain 
et  al.  (1993)  described  a  pioneer  telemedicine  outreach  effort  conducted  between 
Tripler  Army  Medical  Center,  Hawaii,  and  outlying  small  islands  in  the 
Kwajalein  Atoll.  The  patient  population  is  approximately  3,000  American 
workers  under  contract  to  the  DoD.  TRIPLER  medical  center  provided  remote 
VTC  consults  for  59  cases  during  the  period  of  the  study.  Multiple  specialties 
were  used,  and  preliminary  data  indicated  these  consults  helped  avoid  costly 
evacuations  of  patients.  Fifteen  evacuations  were  avoided,  saving  an  estimated 
$2000  per  trip.  Prior  to  the  implementation  of  telemedicine  in  the  area,  specialty 
diagnosis  and  care  often  required  a  costly  evacuation  to  Honolulu.  Tripler 
physicians  reported  initial  success  in  the  first  year  of  operation  using  telemedicine 
in  7  different  specialty  fields:  dermatology,  orthopedics,  radiology, 
ophthalmology,  urology,  pediatrics,  and  physical  therapy.  The  authors  suggest 
that  the  telemedicine  system  will  be  cost  effective. 

Summary 

The  cost-effectiveness  of  a  telemedicine  system  may  be  specific  to  the 
treatment  setting.  Evaluating  the  costs  of  implementing  telemedicine  in  deployed 
treatment  sites  (for  example,  different  ship  types)  would  permit  us  to  develop  an 
investment  strategy  for  the  effective  adaptation  of  telemedicine  capabilities  across 
sites.  In  addition,  studies  that  track  the  path  of  medical  interventions  from 
presentation  to  solution  can  help  develop  straightforward  criteria  for  correlating 
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the  level  of  technology  with  the  level  of  care  needed  at  various  treatment  sites  for 
a  maximum  return  on  investment. 

4.4  Integration  or  Outcome  Models 

Integration  models  assess  clinical  outcomes  in  medical  encounters  in 
conventional  and  telemedicine  settings.  Outcomes  can  be  assessed  within  a 
particular  specialty  or  condition,  or  tracked  over  an  entire  system  to  indicate  the 
efficacy  of  telemedicine  for  diagnosis  and  treatment. 

Roine,  Ohinmaa,  and  Hailey  (2001)  completed  a  comprehensive  review  of 
evaluation  models  in  the  telemedicine  literature  according  to  strict  criteria  for 
inclusion.  Articles  reviewed  had  to  assess  outcomes  of  the  use  of  telemedicine  in 
terms  of  administrative  changes,  patient  outcomes,  or  economic  effects.  In 
addition,  studies  had  to  compare  the  use  of  telemedicine  with  conventional 
alternatives  in  a  scientifically  valid  manner.  Thus,  articles  that  described  specific 
telemedicine  initiatives  or  assessed  the  feasibility  of  a  particular  application  were 
omitted  from  review.  Roine  and  colleagues  began  with  1,224  studies  between 
1966  and  2000.  Fifty  (50)  met  criteria  for  review.  Most  of  these  (34)  included 
clinical  outcomes,  while  the  remaining  sixteen  (16)  were  mainly  cost  analyses. 
Strong  evidence  for  clinical  effectiveness  was  found  for  specialties  such  as 
teleradiology,  teleneurosurgery,  and  telepsychiatry.  Teleradiology  was  supported 
as  a  cost  savings  speciality,  but  more  evidence  is  need  to  evaluate  other  specialties 
for  cost  savings.  The  authors  concluded  that,  while  telemedicine  has 
demonstrated  success  in  some  specialties  and  applications,  scientific  data  for 
appropriate  evaluation  of  telemedicine  continue  to  be  sparse.  Therefore,  broader 
applications  of  telemedicine  should  be  restricted  at  this  time. 

One  of  two  structural  models  for  evaluation  that  Grigsby  et  al.  (1995) 
proposed  evaluates  the  diagnostic  effectiveness  of  telemedicine.  This  model 
focuses  on  the  types  of  conditions  that  might  be  considered  productive  indicators 
of  clinical  effectiveness.  Accurate  diagnoses  in  conditions  that  have  relatively 
high  incidence,  are  moderately  difficult  to  diagnose,  and  that  pose  a  significant 
risk/benefit  and  relief  from  suffering  are  good  indicators  of  the  effectiveness  of 
telemedicine  compared  to  face-to-face  consults.  The  authors  advocated  assessing 
the  effectiveness  of  telemedicine  for  a  limited  number  of  diseases  and  relative 
access  to  care  in  order  to  evaluate  medical  effectiveness.  Their  companion  model, 
proposed  for  use  in  conjunction  with  the  outcome  analysis,  evaluates  telemedicine 
usage  by  task  (Section  4.1). 

Military  Integration/Outcome  Studies 

Burgess  et  al.  (1999)  assessed  the  structural  model  used  in  the  application 
of  telemedicine  to  otolaryngology  in  a  research  and  residency  training  setting  in 
the  Department  of  Surgery  at  Tripler  Army  Medical  Center.  Both  military 
personnel  and  civilians  in  the  Pacific  Island  Nations  are  referred  to  Tripler  Army 
Medical  Center  for  otolaryngological  problems.  TRIPLER’s  approach  to 
developing  telemedicine  programs  was  reviewed,  beginning  with  needs 
assessment,  evaluation  of  appropriate  telemedicine  technologies  and  deployment 
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of  new  systems.  The  authors  of  this  study  advocated  using  VTC  and  store-and- 
forward  technology  to  increase  patient  access  to  specialty  clinics,  reduce  costs 
incurred  from  remote  off-island  treatment  referrals,  reduce  the  isolation  of 
remotely  based  providers,  and  assist  in  medical  training.  They  also  identified  the 
need  for  specialty  speech  care  in  remote  clinics  and  are  in  the  process  of  studying 
the  effects  of  live  versus  VTC  speech  therapy  on  outcomes  for  patients.  If  there 
are  no  substantial  differences,  telespeech  therapy  could  be  deployed  as  the 
standard  of  care. 

Navein,  Hagmann,  and  Ellis  (1997)  assessed  outcomes  of  telemedicine 
consults  in  Army  peacekeeping  operations  in  Macedonia.  Referring  and 
consulting  physicians  completed  questionnaires,  and  follow-up  interviews  of 
referring  physicians  were  conducted.  The  study  found  that  the  availability  of 
telemedicine  reduced  the  number  of  evacuations,  saving  $180,000  to  $300,000 
and  approximately  30  workdays  for  the  period  January  1994  to  April  1995. 
Telemedicine  consults  also  were  found  to  affect  treatment  in  30  of  the  47  cases 
studied.  Provider  satisfaction  was  rated  high  (89%),  with  the  providers  most 
familiar  with  telemedicine  technologies  the  most  likely  to  utilize  them.  Sixty 
percent  of  consults  led  to  changes  in  case  management,  including  treatment, 
mostly  in  dermatology  cases.  Twenty  six  percent  of  evacuation  decisions  were 
affected  by  the  use  of  telemedicine.  The  authors  advocated  further  application  of 
telemedicine,  supported  by  user  training  and  equipment  upkeep.  Pretraining  of 
providers  on  the  telemedicine  systems  was  thought  to  be  critical  to  success  in  the 
field. 

Walters  (1996)  assessed  the  clinical  utility  of  telemedicine  in  consultations 
from  deployed  military  medical  units  in  Somalia,  Haiti,  Croatia,  and  Macedonia 
during  the  period  February  1993  to  March  1995.  A  retrospective  case  review  was 
performed  by  physicians  to  assess  the  severity  of  illness,  the  communication 
modality  utilized  for  the  consultation,  and  changes  in  diagnosis,  treatment,  or  duty 
status  of  the  patient  due  to  the  consultation.  Diagnoses  were  affected  in  30%  of 
cases,  treatment  in  32%,  and  overall  patient  status  in  70%  of  consults.  This 
demonstrates  the  feasibility  of  deployment  telemedicine  in  the  field.  Noting  that 
this  retrospective  study  was  based  on  data  not  originally  designed  to  be  evaluated 
(an  important  factor  in  many  military  studies),  the  author  recommended 
establishing  a  formal  structure  of  oversight  to  ensure  adequate  training, 
standardized  procedures,  and  maintenance  of  equipment. 

Summary 

Telemedicine  has  been  integrated  successfully  in  several  specialties,  most 
notably  radiology  and  ENT.  In  fixed  treatment  settings  (i.e.  brick  and  mortar 
facilities),  telemedicine  can  increase  remote  access  to  care,  reduce  costs  incurred 
from  remote  treatment  referrals,  reduce  the  isolation  of  remotely  based  providers, 
and  assist  in  medical  training.  In  deployed  settings,  telemedicine  technologies 
can  reduce  the  number  of  evacuations  and  lower  the  cost  of  providing  an 
appropriate  level  of  care.  A  model  predicting  the  conditions  where  telemedicine 
would  be  most  successfully  integrated  could  measure  potential  impact  in  deployed 
as  well  as  fixed  treatment  settings. 
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4.5  Cost  Analyses 


Cost  analyses  are  understandably  the  most  prevalent  in  the  telemedicine 
literature,  especially  in  the  early  days  of  widespread  adoption  and 
implementation.  Most  focus  on  projected  or  real-time  costs  for  installation, 
maintenance,  and  usage  of  telemedicine  equipment.  Savings  or  costs  avoided 
may  also  be  considered.  As  existing  telemedicine  systems  age  or  technology 
changes,  cost  analyses  can  also  be  performed  to  assess  cost  of  equipment 
replacement  and  retraining  of  users. 

McCue  et  al.  (2000)  conducted  a  retrospective  analysis  of  3  years  of  data 
to  measure  the  cost  per  visit  of  VTC  telecardiology  services  and  non-telemedicine 
cardiology  services  for  patients  from  a  Virginia  correctional  setting.  Differences 
in  costs/savings  were  measured  in  these  categories: 

•  Medical  (tests,  facility  fees) 

•  Labor 

•  Nonlabor  (such  as  T-l  lines) 

•  Transportation 

•  Total  annual  costs 

The  study  found  that  telecardiology  can  generate  savings  with  increased 
utilization  of  services.  The  feature  of  telemedicine  that  allows  cost  reduction  over 
time  is  fixed  operational  costs. 

A.E.  Cameron,  Bashshur,  Halbritter,  Johnson,  and  J.W.  Cameron  (1998) 
suggested  a  simulation  model  for  projecting  the  financial  performance  of  a  mature 
telemedicine  system,  even  in  the  early  stages  of  development,  and  with  limited 
empirical  data.  A  computer  simulation  for  estimating  the  financial  patterns  of  a 
hub-and-spokes  telemedicine  network  was  developed  and  tested  in  the  MDTV 
program  in  West  Virginia.  The  model  simulated  real-world  situations  in  which 
decisions  are  made  within  model  variables  and  parameters.  This  paper  develops  a 
model  for  projecting  telemedicine  savings  relative  to  in-person  care,  as  empirical 
data  on  this  question  are  lacking.  The  model  shows  substantial  savings  are 
possible,  but  results  vary  with  the  assumptions  of  the  model.  A  critical 
assumption  is  frequency  of  use:  more  use  of  a  telemedicine  system  leads  to  more 
savings.  Telemedicine  programs  therefore  need  to  reduce  barriers  to  telemedicine 
use  to  enhance  the  financial  success  of  telemedicine. 

Brunicardi  (1998)  assessed  actual  costs  in  a  pilot  telemedicine  program 
that  connected  a  university  medical  center  with  the  medical  facilities  in  the  Ohio 
prison  system.  Costs  associated  with  the  use  of  two-way  interactive  video  to 
deliver  health  services  were  compared  with  those  incurred  without  the  use  of 
telemedicine.  In  addition  to  direct  costs  associated  with  telemedicine  equipment 
and  operations,  the  study  stressed  the  need  for  assessment  of  indirect  savings  in 
transportation,  correction  officer  time,  and  security  surveillance  when  taking 
prisoners  into  the  community  for  healthcare.  Straightforward  cost  analysis 
obscures  costs  that  are  avoided  in  telemedicine  health  care.  Assessing  both  costs 
incurred  and  costs  avoided  resulted  in  modest  per- visit  and  quarterly  savings  in 
the  pilot  prison  telemedicine  project.  Zincone,  Doty,  and  Balch  (1997)  performed 
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a  similar  empirical  analysis  of  costs  in  North  Carolina  State  Prison,  with  similar 
recommendations.  This  paper  proposed  a  method  for  determining  the  “break  even 
point”  where  the  telemedicine  service  paid  for  itself.  In  this  case,  the  system 
reached  “break-even”  after  two  years  of  operation.  Importantly,  the  “break-even” 
point  can  be  affected  by  the  rate  of  usage  or  excessive  costs  for  telemedicine 
equipment  or  personnel. 

An  analysis  of  structures  for  evaluation  by  Sisk  and  Sanders  (1998) 
addressed  specific  challenges  to  telemedicine  evaluators:  joint  costs,  multiple 
uses,  expansion  of  use  beyond  original  intentions,  and  technological  innovation 
that  alters  telemedicine  systems  even  while  they  are  being  evaluated.  This 
framework  for  the  economic  evaluation  of  telemedicine  would  assess  three 
criteria:  quality,  accessibility,  and  efficiency  of  care. 

Military  Cost  Analyses 

Stoloff,  Garcia,  Thomason,  and  Shia  (1998)  estimated  costs  and  projected 
equipment  and  bandwidth  requirements  for  the  implementation  of  telemedicine 
aboard  more  than  300  Navy  ships.  The  study  assessed  patient  visits  over  a  one- 
year  period  in  order  to  estimate  both  the  potential  savings  and  the  clinical  impact 
that  of  a  telemedicine  system  might  have  enabled  over  that  period.  An  additional 
survey  of  shipboard  medical  personnel  indicated  a  possible  substantial  savings 
from  a  lower  incidence  of  medevacs  through  the  use  of  telemedicine  consults. 

The  providers  determined  that  telemedicine  would  have  enhanced  care  in  about 
two  thirds  of  all  consults.  Modeling  revealed  that  store  and  forward  telemedicine 
would  be  cost  effective  on  all  ships  and  submarines  and  VTC  only  for  carriers  and 
amphibious  ships.  Teleradiology  would  be  cost  effective  only  for  carriers. 
Seventeen  percent  of  medevacs  would  have  been  prevented  with  telemedicine. 

The  study  concluded  that  the  use  of  email,  fax,  and  the  Internet  would  be  cost 
effective  on  all  types  of  Navy  ships.  Larger  ships  could  support  VTC  and  some 
digital  diagnostic  instruments.  Giving  shipboard  medical  personnel  access  to  ship 
communication  capabilities  would  cost  less  than  relying  on  a  commercial  satellite. 
Overall,  telemedicine  would  improve  quality  of  care  and  help  reduce  the 
incidence  of  medevacs. 

Garcia  and  Stoloff  (1997)  estimated  the  peacetime  need  for  telemedicine 
for  ships  at  sea  and  assessed  costs  and  benefits  of  several  types  of  applications. 
Four  telemedicine  technologies  were  analyzed:  email,  Internet,  VTC,  and  a 
digitizing  scanner  added  to  existing  x-ray  equipment.  The  incidence  of  medevacs 
was  reconstructed  using  a  self-administered  survey  of  shipboard  medical 
departments.  A  panel  of  physicians  and  IDCs  considered  8,000  medical 
encounters  from  the  Snap  Automated  Medical  System  (SAMS)  to  determine 
which  cases  would  have  benefited  from  telemedicine  consults.  The  study  found 
that  the  net  savings  of  telemedicine  over  traditional  medicine  was  relatively  small, 
some  $35,000  per  ship.  However,  another  finding  was  that  implementing 
telemedicine  in  the  Fleet  could  improve  the  quality  of  life  for  sailors  and  promote 
Fleet  readiness.  The  authors  advocated  assigning  more  bandwidth  to  shipboard 
medical  departments. 
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Summary 

While  of  course  the  military  mission  to  provide  medical  care  to  troops 
takes  precedence  over  bottom  line  considerations  of  cost,  cost  analysis  is 
important  in  evaluating  telemedicine  as  an  alternative  to  conventional  methods  of 
care  to  maximize  return  on  investment.  Cost  analyses  in  isolation  from 
assessments  of  other  criteria  are  too  limited  to  provide  a  good  prediction  of  return 
on  investment,  however.  A  straightforward  investment  strategy  that  takes  into 
account  treatment  setting,  level  of  care,  level  of  telemedicine  suite,  and  other 
factors  in  addition  to  costs  should  be  developed  so  that  the  appropriate  level  of 
telemedicine  can  be  implemented  in  each  setting. 

4.6  Evaluations  of  Participant  Satisfaction 

Evaluations  of  participant  satisfaction  focus  on  the  patient’s  level  of 
comfort  with  the  telemedicine  medical  transaction,  or  on  provider  perceptions  of 
and  potential  usage  of  telemedicine.  Patient  and  provider  satisfaction  studies  can 
be  used  to  predict  telemedicine  usage.  In  general,  findings  from  satisfaction 
evaluations  have  been  quite  positive,  perhaps  because  the  study  participants  were 
those  already  inclined  to  accept  teleconsults.  Telemedicine  studies  often  note  that 
experience  with  or  exposure  to  teleconsults  is  associated  with  positive  attitudes 
toward  telemedicine  (Walters  1996,  Karp  et.  al  2000).  Interviewing  only  those 
who  had  already  decided  to  use  telemedicine  as  a  provider  or  patient  excludes 
those  who  may  have  decided  not  to  use  telemedicine  or  those  who  were 
undecided.  Thus,  a  true  random  sample  of  patients  and  providers  without 
selecting  for  telemedicine  experience  might  reveal  more  overall  skepticism. 

Mair  and  Whitten  (2000)  identified  32  empirical  patient  satisfaction 
studies  for  review.  The  review  concluded  that  studies  to  date  have  not  accounted 
for  the  underlying  reasons  for  patient  satisfaction  or  dissatisfaction.  Few  studies 
used  strong  methodology  or  carefully  developed  questions  to  address  important 
telemedicine  issues.  In  addition,  methodological  deficiencies  may  make  results  in 
many  studies  nongeneralizable.  Methodologically  rigorous  research  into  patient 
and  provider  attitudes  toward  telemedicine  is  recommended. 

Sixsmith  (2000)  evaluated  data  on  patient  satisfaction  with  technologies 
designed  to  create  a  home  monitoring  system  for  geriatric  patients.  A  field  study 
of  22  subjects  between  ages  60  and  85  tested  an  intelligent  home  monitoring 
system.  Sensors  measured  the  level  of  activity  over  a  given  period  of  time; 
behaviors  such  as  time  out  of  bed  or  use  of  the  refrigerator;  room  temperature; 
and  unusual  activity  that  could  indicate  a  fall  or  other  distress.  Although  a  large 
number  of  false  alerts  were  generated  during  the  study  period,  and  no  real 
emergencies  occurred,  participants  reported  feeling  safer.  Thus,  patients  with  a 
large  personal  stake  in  being  perceived  as  autonomous  and  independent  were 
willing  to  work  with  deficiencies  in  current  technologies  in  order  to  receive 
perceived  benefits,  a  factor  that  may  prove  helpful  in  predicting  help-seeking 
behaviors  and  the  usage  of  telemedicine  in  settings  where  autonomy  and 
competence  are  highly  valued. 
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At  least  two  studies  found  provider  dissatisfaction  with  telemedicine. 

May  et  al.  (2001)  analyzed  structural  changes  in  a  mental  health  clinic  after  the 
introduction  of  a  videophone  for  psychiatric  consultations.  Interviews  with 
mental  health  practitioners,  general  practitioners,  and  patients  suggested  that 
telemedicine  had  limitations  for  some  patients  (e.g.,  extreme  anxiety)  and  that 
specialists  were  most  wary  of  its  limitations  for  care.  Most  providers  believed 
experience  with  the  system  would  allow  them  to  adapt  to  provide  effective  care. 
However,  after  an  initial  period  of  enthusiasm,  providers  increasingly  and  actively 
resisted  the  technology.  Practitioners  attempted  to  augment  their  established 
models  of  practice,  rather  than  using  the  technology  to  create  new  models. 
Importantly,  the  system  was  ultimately  rejected,  not  because  of  technical 
limitations,  but  because  users  could  not  make  it  conform  to  the  medical 
transaction  model  already  in  use. 

In  another  study  evaluating  structural  models  for  telemedicine  usage,  May 
and  Ellis  (2001)  found  that  staff  perceptions  about  the  purpose,  design,  and 
service  of  a  proposed  telemedicine  clinic  undermined  their  attempt  to  obtain 
funding,  even  as  they  agreed  on  the  need  for  the  project  being  proposed.  While  no 
data  or  outcomes  were  achieved,  this  study  points  out  the  importance  of  user  buy- 
in  and  participant  satisfaction  with  telemedicine  systems  even  in  the  planning 
stages.  May  and  Ellis  conclude  that  human  perceptions  of  what  capabilities  and 
functions  a  system  should  address  is  as  important  to  the  success  of  a  telemedicine 
system  as  the  technology,  itself. 


Military  Patient  or  Provider  Satisfaction  Studies 

Participant  satisfaction  is  often  cited  as  one  factor  among  many  in  military 
studies  of  telemedicine.  The  authors  have  not  found  a  dedicated  study  of 
participant  satisfaction  in  the  military  telemedicine  literature. 

Summary 


Participant  satisfaction  measures  the  human  “return  on  investment”  in  a 
telemedicine  system.  This  is  important  for  military  studies  in  two  key  areas.  First, 
the  self-diagnostic  capabilities  telemedicine  provides  may  increase  help-seeking 
behavior  and  preventive  care  in  settings  where  autonomy  and  competence  are 
highly  valued.  Second,  the  successful  implementation  of  a  technology  depends  in 
large  part  on  the  perception  of  users  that  it  is  useful.  Any  strategy  for  estimating 
the  level  of  telemedicine  needed  at  a  particular  level  of  care  should  take  into 
account  how  the  technology  will  change,  eradicate,  or  clash  with  practices  already 
in  place.  Money  and  personnel  hours  are  wasted  if  technology  is  perceived  as  an 
additional  tasking  rather  than  as  a  tool  for  managing  workload  more  effectively. 

4.7  Human  Factors  Evaluations 

Human  factors  evaluations  overlap  many  of  the  preceding  categories. 
Studies  assess  any  of  the  following: 

•  Human/machine  interaction 
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•  Provider/patient  transactions 

•  User  training  or  development  of  preference  for  one  modality  over  another 

•  Development  and  promotion  of  viable  telemedicine  systems 

A  structural  model  by  Yellowlees  (1997, 1998)  discussed  telepsychiatry  in 
Australia  and  proposed  core  human  factors  principles  in  developing  successful 
telemedicine  systems.  Central  in  this  model  is  a  focus  on  the  provider  perceptions 
of  “ownership”  or  ego  investment  that  drive  the  establishment  and  development 
of  workable  systems.  Decisions  based  on  management  policies,  equipment 
parameters,  patient  need,  or  medical  specialty  are  not  as  likely  to  promote  a 
coherent  and  viable  telemedicine  system  as  policy  built  around  the  needs  and 
interests  of  early  adopters.  Care  providers  and  users  who  enthusiastically  adopt 
telemedicine  applications  work  actively  to  support,  promote,  troubleshoot,  and 
otherwise  strategize  the  ongoing  development  of  the  overall  system.  These  early 
adopters  provide  a  solid,  tangible  base  from  which  to  expand  implementation  of 
telemedicine. 


Human  Factors  in  Military  Studies 

Human  factors  tend  to  be  assessed  as  part  of  a  larger  cluster  of  factors  in 
military  telemedicine  studies.  This  suggests  that  the  organizational  aspects  of 
telemedicine  in  a  military  setting  could  be  explored  in  more  detail. 

A  structural  model  proposed  by  Eliasson  and  Poropatich  (1998)  suggested 
a  constellation  of  quality  management  (QM)  tools  to  promote  performance 
improvement  (PI)  in  telemedicine  systems.  Five  categories  of  telemedicine 
management  at  WRAMC  are  described  and  proposed  as  guidelines  for  other 
telemedicine  programs: 

•  Licensing  and  credentialing 

•  Security  and  privacy 

•  Consent 

•  Peer  review 

•  Tailored  efforts  to  improve  performance 

Summary 

Any  model  for  calculating  the  level  of  telemedicine  to  implement  in  a  given 
treatment  setting  should  include  an  assessment  of  human  factors.  Provider  investment  in 
and  ease  with  telemedicine  is  critical  for  program  success:  user  understanding  of  a 
system’s  capabilities  can  determine  whether  or  not  a  telemedicine  system  is  utilized  to  its 
fullest  capacity.  Assessing  human/machine  interaction,  provider/patient  transactions,  and 
user  training  can  also  help  insure  what  an  Army  researcher  recently  called  "the  critical 
issue":  that  the  technologies  implemented  in  a  given  setting  assist  rather  than  encumber 
medical  personnel  in  performing  their  duties  (Reed  2002). 
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5.0  Recommendations  for  Future  Studies 

As  telecommunications  technologies  evolve  and  proliferate,  they  are  becoming 
more  and  more  an  unremarked  backdrop  to  everyday  healthcare  practices  in  both  military 
and  civilian  settings.  Medical  training  for  care  providers  is  increasingly  specialized, 
allowing  personnel  the  ability  to  utilize  the  more  sophisticated  capabilities  of 
telemedicine  systems  across  a  continuum  of  care.  Thus,  research  that  assesses  how 
telemedicine  should  be  implemented,  how  much  it  may  cost,  who  uses  it,  what  treatment 
outcomes  it  enables,  or  even  a  combination  of  these  categories,  may  have  run  its  course. 
The  focus  of  current  telemedicine  research  should  now  shift  from  asking  “How  do 
telemedicine  configurations  work  in  a  military  setting?”  to  “What  level  of  telemedicine 
should  be  implemented  to  achieve  the  maximum  return  on  our  investment?” 

Military  telemedicine  systems  vary  widely,  both  in  treatment  facility  and  in  the 
clusters  of  technologies  they  utilize  to  deliver  care.  Levels  of  telemedicine  capabilities 
do  not  easily  correlate  to  levels  of  care:  several  different  telemedicine  suites  may  be 
implemented  in  a  single  echelon.  The  continuum  of  treatment  sites  ranges  from  highly 
mobile  to  wholly  fixed  sites,  with  different  combinations  and  levels  of  telemedicine 
technologies  available  for  implementation  within  different  levels  of  care.  Many  times, 
what  drives  the  success  of  the  implementation  of  telemedicine  is  not  the  setting  or  level 
of  care,  but  rather  the  specialized  training  and  investment  of  the  personnel  using  it.  (See 
Figures  2  and  3). 
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Figure  2  Telemedicine  Technology  Suites 
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Figure  3  Technical  Capabilities  Across  Treatment  Sites 


Levels  of  Care  and  Associated  Telecommunications  Capabilities 
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With  complex  systems  and  variable  treatment  sites,  individual  studies  can  focus 
on  only  one  or  two  of  the  multiple  components  operating  within  a  telemedicine  system. 
Even  a  series  of  narrowly-focused  short-term  studies  will  not  produce  an  accurate  picture 
of  how  well  the  overall  system  is  functioning.  However,  a  series  of  narrowly-defined 
short-term  studies  can  begin  to  produce  an  accurate  picture  of  how  levels  of  telemedicine 
correlate  to  levels  of  care.  These  results  can  be  codified  to  develop  a  model  for 
determining  the  type  of  telemedicine  suite  to  implement  across  variable  treatment  sites, 
levels  of  connectivity,  levels  of  care,  and  patient  needs.  Such  a  model  can  help  ensure 
that  funds  are  invested  in  the  most  effective  way  to  serve  our  forces.  Two  suggestions  for 
baseline  studies  to  develop  a  workable  model  follow. 

5.1  A  Technology-Based  Approach  for  Classifying  Telemedicine  Systems 

The  emerging  focus  of  Navy  telemedicine  studies  should  be  to  develop  a 
classification  system  for  assessing  telemedicine  systems  based  on  technical  capabilities. 
This  could  be  accomplished  with  a  series  of  short-term  studies  that  test  the  utility  of 
telemedicine  in  a  particular  setting.  This  approach  would  isolate  a  small  telemedicine 
system  in  which  multiple  components  might  be  assessed  as  parts  of  a  functioning  unit. 
For  example,  military  clinics  might  be  classified  by  their  telemedicine  capabilities.  A 
continuum  of  capabilities  could  be  established,  with  fully  equipped  telemedicine  facilities 
at  one  end  and  clinics  with  minimal  telecommunications  resources  at  the  other.  Each 
class  of  clinic  could  be  assessed  based  on  outcome  measures  such  as  clinical  impact, 
operational  efficiency  and  integration  of  various  components  of  the  system  with  overall 
clinic  functioning. 

The  simplest  method  for  classification  of  clinics  is  to  rank  their  telemedicine 
capabilities  from  basic  to  advanced  technologies.  Thus,  the  basic  telemedicine  suite  for  a 
clinic  might  consist  of  telephone  and  fax  for  remote  consultations.  More  advanced 
classes  of  telemedicine  suites  would  consist  of  added  technologies  such  as  email, 

Internet,  computed  radiography,  various  other  diagnostic  videoscoping  procedures  (e.g., 
ENT  otoscope)  and  live  videoconferencing.  Important  variables  such  as  medical  staffing 
at  each  clinic,  patient  population,  and  geographic  location  could  also  be  assessed  and 
isolated  statistically  to  permit  a  controlled  comparison  of  clinics  on  the  outcome 
measures  described  above.  A  technology-based  approach  for  classifying  and  evaluating 
functioning  telemedicine  systems  could  be  developed  from  the  findings  of  these  studies. 

5.2  Assessing  a  Pilot  Telemedicine  System  or  Site 

A  single  pilot  system  or  site  could  be  used  to  test  structural  models  for 
assessing  Navy  telemedicine  as  a  complex  healthcare  delivery  and  information 
system.  A  systemic  evaluation  of  telemedicine  in  a  pilot  system  or  “node”  would 
allow  researchers  to  analyze  how  the  seven  evaluative  categories  discussed  in  this 
report  function  in  relation  to  each  other  and  to  the  larger  system. 

For  example,  the  Medical  Data  Surveillance  System  (MDSS),  an  R  &  D 
product  currently  under  development  that  looks  for  trends  in  patient  encounter 
data,  could  be  evaluated  as  a  closed  “telemedicine”  system.  Each  of  the  seven 
components  for  evaluation  discussed  in  this  technical  report  can  be  assessed 
within  the  MDSS  system:  Task  domains,  Tools,  Settings,  Integration,  Costs, 
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Participant  satisfaction,  and  Human  factors.  These  components  can  be  assessed  in 
two  ways.  First,  each  component  can  be  evaluated  as  part  of  the  MDSS  system  as 
a  whole  to  determine  how  effective  individual  factors  operate  within  the  larger 
system.  Second,  the  interoperability  of  these  seven  components  can  be  assessed 
to  determine  the  cumulative  or  systemic  effect  of  all  seven  components. 

Following  are  some  issues  to  be  addressed: 

•  How  efficient  is  the  overall  system? 

•  How  does  each  component  interact  with  and  affect  each  of  the  others? 

•  How  does  each  component  fit  into  the  system  as  a  whole? 

•  If  one  or  more  components  is  not  functioning  at  full  capacity,  how  does 
that  effect  the  efficiency  of  the  overall  system? 

•  Are  any  of  these  elements  working  against  each  other? 

Alternatively,  a  specific  site  could  be  assessed  as  one  “node”  in  a  complex 
information  system  that  gives  us  information  about  how  the  larger  system 
functions.  In  particular,  researchers  would  analyze  the  procedures  and 
organizational  structures  for  proposing,  implementing,  assessing,  and  “growing” 
telemedicine  applications  within  the  test  site. 

Focus  groups  could  develop  a  framework  of  evaluative  criteria  applicable 
to  the  test  site.  The  framework  of  evaluative  criteria  that  emerges  from  focus 
group  discussions  would  be  used  to  assess  telemedicine  studies  already  completed 
or  in  progress  at  the  test  site,  as  well  as  develop  future  projects  and  assess  the 
growth  and  health  of  the  telemedicine  system  at  the  test  site.  Some  components 
to  consider  in  each  of  the  seven  evaluation  categories  are  as  follows: 

•  Assessment  of  current  research/efificacy  of  telemedicine  at  the  test  site 

•  Criteria  forjudging  proposed  research  or  areas  to  implement  telemedicine 

•  Criteria  for  assessing  which  projects  should  get  the  green  light 

•  Structures  and  procedures  for  assessing  work  in  progress  (projects  as  they 
develop) 

•  Structures  for  generating  written  reports  of  work  in  progress 

•  Structures  for  promoting  completed  or  successful  projects  to  the  larger 
telemedicine  community 

•  Ongoing  assessment  of  the  overall  health  of  the  telemedicine  project 

Researchers  can  craft  this  systemic  evaluation  as  an  analysis  of 
organization  (a  structural  analysis  with  no  data);  a  set  of  empirical  studies  of  the 
components  at  the  test  site;  or  a  combination  of  the  two,  based  on  available 
funding  and  time  frames. 

Appendices  A  and  B  outline  possible  structures  for  preliminary  focus 
group  discussions  and  short-term  studies  that  can  be  used  to  determine  criteria  for 
a  working  model  for  assessing  telemedicine  as  a  complex  system.  Preliminary 
studies  are  the  first  step  in  developing  a  model  to  determine  which  levels  of 
telemedicine  correlate  to  specific  levels  of  care. 
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6.0  Conclusion 


An  analysis  of  civilian  and  military  models  for  evaluating  telemedicine  indicates 
that  a  straightforward  model  for  calculating  military  telemedicine’s  return  on  investment 
should  be  developed.  There  is  currently  no  workable  model  for  evaluating  telemedicine 
as  a  system,  either  in  military  or  civilian  research.  While  military  telemedicine 
circumvents  some  problems  faced  in  civilian  systems,  issues  such  as  wait  and  queue, 
connectivity,  continuum  of  care  across  treatment  sites,  and  care  needs  in  deployed 
platforms  make  it  difficult  to  measure  the  effectiveness  of  military  telemedicine  at 
different  echelons  of  care.  A  model  should  be  developed  that  dictates  the  appropriate 
telemedicine  suite  to  implement  for  a  given  population  at  risk  across  varying  treatment 
facilities  so  that  the  maximum  return  on  investment  can  be  achieved. 

As  Navy  telemedicine  responds  to  emerging  technologies  and  shifts  in  care  needs, 
it  is  likely  that  current  concerns  such  as  bandwidth  requirements  or  image  quality  will 
become  non-issues,  new  applications  will  arise,  and  care  will  extend  to  more  remote 
treatment  populations.  A  method  to  evaluate  the  complex  structure  and  organization  of 
telemedicine  should  be  developed  to  provide  consistent  parameters  within  which  to 
interpret  and  utilize  the  system  as  it  matures  and  further  evolves.  With  troops  deployed  in 
remote,  austere,  and  relatively  inaccessible  environments,  technical,  human,  and 
organizational  components  across  the  system  must  enable  the  free  flow  of  information  to 
assure  quality  of  care.  Evaluating  the  structure  of  the  overall  system,  as  well  as  its 
components,  can  help  realize  the  potential  for  interoperability  and  enable  quick  response 
to  current  defense  needs. 
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Appendix  A 

Preliminary  Focus  Group  Discussions 

A  preliminary  study  of  criteria  to  include  in  a  model  for  assessing  a  pilot  telemedicine  system  or 
site  can  be  conducted  with  focus  groups.  Initial  focus  groups  would  be  comprised  of  the  early 
adopters  and  policy  setters  who  implement  telemedicine  at  the  treatment  site  or  make  strategic 
decisions  about  the  management  of  telemedicine  in  the  Navy.  This  composition  of  groups  would 
allow  feedback  from  both  on-site  users  and  administrators.  Participants  would  be  provided  with 
the  set  of  evaluation  categories  described  in  this  report  and  asked  to  rank  how  important  each  of 
these  categories  is  in  evaluating  Navy  telemedicine.  Categories,  or  items  within  categories, 
could  be  added  or  deleted.  Focus  groups  composed  of  both  on-site  users  and  administrators 
would  follow  their  ranking  and  assessment  with  a  discussion  of  recommendations  for  evaluating 
each  category. 


A  working  model  for  assessing  telemedicine  on  a  structural  or  systemic  level  would  be 
developed  from  consensus  reached  in  these  preliminary  focus  group  discussions.  Further  studies 
(as  recommended  in  Appendix  C)  could  assess  one  or  more  of  these  criteria  as  components 
within  the  overall  framework  of  Navy  telemedicine,  or  evaluate  how  the  different  components 
work  in  relation  to  each  other  and  the  system  as  a  whole. 


Overview  of  Preliminary  Focus  Group  Discussions 


Step  1 

Early  telemedicine  adopters  (providers)  and  policy  makers  (administrators)  rank 
criteria  set  out  by  relevant  models  of  evaluation  in  the  telemedicine  literature. 

Step  2 

Focus  group  results  are  formed  into  a  framework  of  evaluative  criteria  applicable  to 
Navy  telemedicine. 

Step  3 

The  framework  of  evaluative  criteria  that  emerges  from  preliminary  focus  groups 
will  be  used  to  develop  further  studies,  as  described  in  Section  5.0  of  this  report. 
These  studies  will  be  used,  not  only  to  assess  the  components  of  Navy 
telemedicine,  but  also  to  refine  the  model  for  evaluating  Navy  telemedicine  as  a 
healthcare  system. 
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Appendix  B 

Categories  for  Continuing  Short-Term  Studies 

Appendix  A  recommends  a  start-up  study  to  develop  criteria  for  evaluating  Navy  telemedicine  as 
a  complex  healthcare  system.  A  systemic  evaluation  model  could  be  developed  to  provide  a 
framework  for  interpreting  current  Navy  telemedicine  within  system-wide  goals  and  concerns. 
Following  are  recommendations  for  studies  in  each  of  the  categories  described  in  this  report. 

Task  Domains 

To  date,  NHRC  evaluations  of  telemedicine  have  focused  primarily  on  applications  by  specialty, 
treatment  site,  or  clinical  outcomes.  A  task-based,  process-oriented  model  could  delineate  the 
categories  of  processes  or  tasks  involved  in  Navy  applications  of  telemedicine  across  specialties. 
This  would  allow  the  evaluation  of  Navy  telemedicine  as  a  system  of  processes  or  tasks,  rather 
than  its  integration  into  discrete  specialties,  such  as  Ear/Nose/Throat  (ENT)  or  radiology. 

Clinical  outcomes  for  diagnosis  and  treatment  could  also  continue  to  be  assessed  at  the  level  of 
condition,  specialty,  or  treatment  setting. 

Tools  and  Equipment 

Component  interoperability  is  extremely  important  for  field  readiness  and  ongoing  Navy  medical 
research.  A  military  assessment  of  interoperability  would  evaluate  the  functions  of  the  existing 
components  of  military  telemedicine  purchased  from  various  vendors  to  propose  solutions  for 
data  exchange.  Key  areas  of  assessment  would  include: 

•  Standards  or  formats 

•  Operation  and  translation 

•  Effectiveness  across  variations  in  software,  hardware,  user,  and  site. 

Component  interoperability  is  critical  to  ongoing  research  and  field  readiness. 

In  addition,  a  human  factors  usability  evaluation  could  assess  the  use  and  effectiveness  of 
telemedicine  equipment  already  purchased  and  in  operation.  Criteria  to  consider  might  include 
the  following: 

•  User  needs 

•  User  satisfaction 

•  Rate  of  use  of  equipment 

•  Perceived  limitations  of  the  existing  system 

•  Reliability  of  current  equipment  (reflected  in  clinical  outcomes) 

Commercially  available  equipment  could  also  be  assessed  utilizing  criteria  developed  from  user 
surveys.  A  model  for  evaluating  both  existing  and  commercially  procurable  equipment  could  be 
designed  from  user  survey  results  and  human  factors  usability  criteria,  such  as  technical 
acceptability,  operational  effectiveness,  and  clinical  appropriateness  for  Navy  medical 
applications. 
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13.  SUPPLEMENTARY  NOTES 

14.  ABSTRACT,  (maximum  200  words) 

Problem*  The  ongoing  war  on  terrorism  makes  medical  surveillance  and  remote  access  to  medical  care  for  deployed  forces 
pressing  concerns.  The  potential  for  chemical  and  biological  terrorism  on  U.S.  soil  adds  an  urgency  for  Navy  medical  technologies 
to  respond  to  Homeland  Defense  concerns.  An  assessment  of  Navy  telemedicine  as  a  complex  healthcare  support  system  is 
needed  to  demonstrate  how  current  practices,  training,  equipment,  and  expenditures  measure  up  to  the  emerging  needs  of  the 
Fleet. 

Objective :  This  report  reviews  military  and  civilian  models  for  evaluating  telemedicine  systems  in  order  to  determine  future 
directions  for  Navy  telemedicine  research  within  the  current  funding  environment.  How  can  we  calculate  the  level  of  technology 
to  implement  for  the  most  productive  return  on  investment  across  varying  treatment  settings? 

Approach:  A  literature  review  of  military  and  civilian  telemedicine  was  conducted  in  September  2001  to  determine  what  types  of 
evaluative  models  are  currently  being  advocated  by  telemedicine  practitioners.  A  review  of  current  models  for  evaluating 
telemedicine  yielded  seven  categories  for  evaluation:  tools  and  equipment;  outcomes;  cost;  treatment  settings,  task  domains; 
participant  satisfaction;  and  human  factors.  Using  this  literature  review  as  a  baseline,  the  authors  developed  a  conceptual  model 
for  assessing  the  structure  of  Navy  telemedicine.  The  model  describes  how  each  of  the  seven  components  affect  the 
implementation  of  telemedicine  in  Navy  and  civilian  settings.  Finally,  the  authors  recommend  how  short*  term,  narrowly 'focused 
studies  could  form  the  basis  for  a  model  for  correlating  level  of  technology  suites  to  level  of  care  across  treatment  settings. 

Results:  An  analysis  of  models  for  evaluating  telemedicine  indicates  that  there  is  currently  no  workable  model  for  evaluating 
telemedicine  as  a  complex  and  multi-level  system,  either  in  military  or  civilian  research.  While  military  telemedicine  circumvents 
some  problems  faced  in  civilian  systems,  issues  such  as  wait  and  queue,  connectivity,  continuum  of  care  across  variable 
treatment  sites,  and  healthcare  needs  in  deployed  platforms  make  it  difficult  accurately  to  measure  the  effectiveness  of 
telemedicine  at  different  levels  of  care. 

Conclusions:  A  model  for  evaluating  telemedicine  that  correlates  level  of  technology  to  level  of  care  should  be  developed.  A 
system  of  measurable  criteria  for  determining  the  appropriate  telemedicine  suite  to  implement  for  a  given  population  at  risk 
across  varying  treatment  facilities  needs  to  be  designed  so  that  the  maximum  return  on  investment  can  be  achieved. 
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